Methods for multi-class cost-sensitive learning

ABSTRACT

Methods for multi-class cost-sensitive learning are based on iterative example weighting schemes and solve multi-class cost-sensitive learning problems using a binary classification algorithm. One of the methods works by iteratively applying weighted sampling from an expanded data set, which is obtained by enhancing each example in the original data set with as many data points as there are possible labels for any single instance, using a weighting scheme which gives each labeled example the weight specified as the difference between the average cost on that instance by the averaged hypotheses from the iterations so far and the misclassification cost associated with the label in the labeled example in question. It then calls the component classification algorithm on a modified binary classification problem in which each example is itself already a labeled pair, and its (meta) label is 1 or 0 depending on whether the example weight in the above weighting scheme is positive or negative, respectively. It then finally outputs a classifier hypothesis which is the average of all the hypotheses output in the respective iterations.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to the field of cost-sensitivelearning in the areas of machine learning and data mining and, moreparticularly, to methods for solving multi-class cost-sensitive learningproblems using a binary classification algorithm. This algorithm isbased on techniques of data space expansion and gradient boosting withstochastic ensembles.

2. Background Description

Classification in the presence of varying costs associated withdifferent types of misclassification is important for practicalapplications, including many data mining applications, such as targetedmarketing, fraud and intrusion detection, among others. Classificationis often idealized as a problem where every example is equallyimportant, and the cost of misclassification is always the same. Thereal world is messier. Typically, some examples are much more importantthan others, and the cost of misclassifying in one way differs from thecost of misclassifying in another way. A body of work on this subjecthas become known as cost-sensitive learning, in the areas of machinelearning and data mining.

Research in cost-sensitive learning falls into three main categories.The first category is concerned with making particular classifierlearners cost-sensitive, including methods specific for decision trees(see, for example., U. Knoll, G. Nakhaeizadeh, and B. Tausend,“Cost-sensitive pruning of decision trees”, Proceedings of the EightEuropean Conference on Machine Learning, pp. 383-386, 1994, and J.Bradford, C. Kunz, R. Kohavi, C. Brunk, and C. Brodley, “Pruningdecision trees with misclassification costs”, Proceedings of theEuropean Conference on Machine Learning, pp. 131-136, 1998), neuralnetworks (see, for example, P. Geibel and F. Wysotzki, “Perceptron basedlearning with example dependent and noisy costs”, Proceedings of theTwentieth International Conference on Machine Learning, 2003), andsupport vector machines (see, for example, G. Fumera and F. Roli,“Cost-sensitive learning in support vector machines”, VIII ConvegnoAssociazione Italiana per L'Intelligenza Artificiale, 2002). The secondcategory uses Bayes risk theory to assign each example to its lowestexpected cost class (see, for example, P. Domingos, “MetaCost: A generalmethod for making classifiers cost sensitive”, Proceedings of the FifthInternational Conference on Knowledge Discovery and Data Mining, pp.144-164, ACM Press, 1999, and D. Margineantu, Methods for Cost-SensitiveLearning, PhD thesis, Department of Computer Science, Oregon StateUniversity, Corvallis, 2001). This requires classifiers to output classmembership probabilities and sometimes requires estimating costs (see,B. Zadrozny and C. Elkan, “Learning and making decisions when costs andprobabilities are both unknown”, Proceedings of the SeventhInternational Confernece on Knowledge Discovery and Data Mining, pp.204-213, ACM Press, 2001) (when the costs are unknown at classificationtime). The third category concerns methods that modify the distributionof training examples before applying the classifier learning method, sothat the classifier learned from the modified distribution iscost-sensitive. We call this approach cost-sensitive learning by exampleweighting. Work in this area includes stratification methods (see, forexample, P. Chan and S. Stolfo, “Toward scalable learning withnon-uniform class and cost distributions”, Proceedings of the FourthInternational Conference on Knowledge Discovery and Data Mining, pp.164-168, 1998, and L. Breiman, J. H. Friedman, R. A. Olsen, and C. J.Stone, Classification and Regression Trees, Wadsworth InternationalGroup, 1984) and the costing algorithm (see, for example, B. Zadrozny,J. Langford, and N. Abe, “Cost-sensitive learning by cost-proportionateexample weighting”, Proceedings of the Third IEEE InternationalConference on Data Mining, pp. 435-442, 2003). This approach is verygeneral since it reuses arbitrary classifier learners and does notrequire accurate class probability estimates from the classifier.Empirically this approach attains similar or better cost-minimizationperformance.

Unfortunately, current methods in this category suffer from a majorlimitation: they are well-understood only for two-class problems. In thetwo-class case, it is easy to show that each example should be weightedproportionally to the difference in cost between predicting correctly orincorrectly (see, again, Zadrozny et al., ibid.). However, in themulti-class case there is more than one way in which a classifier canmake a mistake, breaking the application of this simple formula.Heuristics, such as weighting examples by the average misclassificationcost, have been proposed (see, again, Breiman et al., ibid., and theMargineantu thesis, ibid.), but they are not well-motivatedtheoretically and do not seem to work very well in practice whencompared to methods that use Bayes risk minimization (see, again,Domingos, ibid.).

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide a methodfor multi-class cost-sensitive learning based on an example weightingscheme.

According to the invention, the methods are based on example weightingschemes that are derived using two key ideas: 1) data space expansionand 2) gradient boosting with stochastic ensembles. The latter is aformal framework that give rise to a coherent body of methods.

One of the methods of invention, which is based on the idea 1) above,works by repeatedly sampling from the expanded data set, which isobtained by enhancing each example in the original data set with as manydata points as there are possible labels for any single instance. Itthen repeatedly draws sub-sample from this expanded data set usingweighted sampling according to a certain example weighting scheme, inwhich each labeled example is given the weight specified as thedifference between the maximum possible misclassification cost for theinstance in question and the misclassification associated with the labelin the particular labeled example. The example weighting remainsconstant throughout the iterative sampling procedure. It then finallyoutputs a classifier hypothesis which is the average of all thehypotheses output in the respective iterations.

Another one of the methods of invention, which is based on the idea 2)above, works by iteratively applying weighted sampling from the sameexpanded data set, using a different weighting scheme. The weightingscheme of this method gives each labeled example the weight specified asthe difference between the average cost on that instance by the averagedhypotheses from the iterations so far and the misclassification costassociated with the label in the labeled example in question.Emphatically, the weighting changes in every iteration, since it dependson the performance of the averaged hypothesis obtained up to the currentiteration. Additionally, the example weights used in this method can beboth positive and negative, since the label given in any labeled exampledoes not necessarily correspond to the best label for the giveninstance, i.e. the label with the minimum cost, due to the use of dataspace expansion. Negative weights do not admit the use of weightedsampling. The method deals with this problem by calling the componentclassification algorithm on a modified binary classification problem inwhich each example is itself already a labeled pair, and its (meta)label is 1 or 0 depending on whether the example weight in the aboveweighting scheme is positive or negative, respectively.

The results of the methods of invention are obtained by outputting allof the classifier representations obtained through the iterations, andrepresent the average over them. These representations can be arbitraryrepresentations of classifiers, such as decision trees, neural networksand support vector machines, for the problem at hand, such as networkintrusion detection, fraud detection, targeted marketing, credit riskrating, among other things. For example, in the application to networkintrusion detection, each one of these representations could be adecision tree that specifies a set of conditions on various attributesof a network connection event, which together signal certain types ofnetwork intrusion. Such representations can be further applied on a newnetwork connection to output judgment whether or not the connection isto be suspected to be some type of an intrusion attempt with reasonablelikelihood, and decisions can be based on this judgment to determine theappropriate course of action, such as denial of service or probing.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages will be betterunderstood from the following detailed description of a preferredembodiment of the invention with reference to the drawings, in which:

FIG. 1 is a block diagram showing the architecture of the systemimplementing one of the methods according to the invention;

FIG. 2 is a flow chart showing the logic of the method for multi-classcost-sensitive learning implemented on the system shown in FIG. 1;

FIG. 3 is a block diagram showing the architecture of the systemimplementing another one of the methods according to the invention;

FIG. 4 is a flow chart showing the logic of the method for multi-classcost-sensitive learning implemented on the system shown in FIG. 3; and

FIG. 5 is an example of a decision tree to illustrate the processimplemented by the invention.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

We begin by introducing some general concepts and notation we use in therest of the description.

Cost-Sensitive Learning and Related Problems

A popular formulation of the cost-sensitive learning problem is via theuse of a cost matrix. A cost matrix, C(y₁, y₂), specifies how much costis incurred when misclassifying an example labeled y₂ as y₁, and thegoal of a cost-sensitive learning method is to minimize the expectedcost. Zadrozny and Elkan (B. Zadrozny and C. Elkan, “Learning and makingdecisions when costs and probabilities are both unknown”, Proceedings ofthe seventh International Conference on Knowledge Discovery and DataMining, pp. 204-213, ACM Press, 2001) noted that this formulation is notapplicable in situations in which misclassification costs depend onparticular instances, and proposed a more general form of cost function,C(x, y₁ , y₂), that allows dependence on the instance x. Here we adoptthis general formulation, but note that in the reasonable case in whichthe cost is minimized by the true label, we can drop the redundantinformation y₂ and write C(x, y₁) for C(x, y₁, y₂).

Once we allow the costs to depend on each example, it is natural toassume that the costs are generated according to some distribution,along with the examples, which leads to the following formulation. In(multi-class) cost sensitive classification, examples of the form (x,(C(x, y₁), . . . , C(x, y_(k))) are drawn from a distribution D over adomain X×R⁺ ^(k) . (Throughout, we will let k denote |Y|.) Given a setof examples, S=(x_(i), (C(x_(i), y))_(yεY))^(m), the goal is to find aclassifier h:X→{1, . . . , k} which minimizes the expected cost of theclassifier: $\begin{matrix}{\arg\quad{\min\limits_{h}{E_{D}\left\lbrack {C\left( {x,{h(x)}} \right)} \right\rbrack}}} & (1)\end{matrix}$We can assume without loss of generality that the costs are normalizedso that${\forall{x \in {X\quad{\min\limits_{x \in Y}{C\left( {x,y} \right)}}}}} = 0.$Note that with this normalization, the above formulation of cost isequivalent to the common fonnulation in terms of misclassification cost,i.e.,$\min\limits_{h}{E_{D}\left\lbrack {{C\left( {x,{h(x)}} \right)}{I\left( {{h(x)} \neq {\arg\quad{\min\limits_{y}{C\left( {x,y} \right)}}}} \right)}} \right\rbrack}$Nonmally a learning method attempts to do this by minimizing theempirical cost in the given training data, given some hypothesis class

: $\begin{matrix}{\arg\quad{\min\limits_{h \in H}{\sum\limits_{{({x,{\langle{C{({x,y})}}\rangle}_{y \in Y}})} \in S}{C\left( {x,{h(x)}} \right)}}}} & (2)\end{matrix}$We note that we sometimes use the empirical expectation notation, Ê, torefer to the averaged empirical cost, namely${{\hat{E}}_{{({x,{\langle{C{({x,y})}}\rangle}_{y \in Y}})} - S}{C\left( {x,{h(x)}} \right)}} = {\frac{1}{S}{\sum\limits_{{({x,{\langle{C{({x,y})}}\rangle}_{y \in Y}})} \in S}{C\left( {x,{h(x)}} \right)}}}$As a building block of our method, we make use of methods for solvingimportance weighted classification problems, which we define below. Inimportance weighted classification, examples of the form (x, y, c) aredrawn from a distribution D over a domain X×Y×R⁺. Given a set ofexamples S=(x, y, c)^(m), the goal is to find a classifier h:X→Y havingminimum importance-weighted misclassificaton error:$\arg\quad{\min\limits_{h}{E_{{({x,y,c})} - D}{c \cdot {I\left( {{h(x)} \neq y} \right)}}}}$Again, usually, a learning method attempts to meet this goal byminimizing the empirical weighted error in some hypothesis class

: $\begin{matrix}{\arg\quad{\min\limits_{h \in H}{\sum\limits_{{({x,y,c})} \in S}{c \cdot {I\left( {{h(x)} \neq y} \right)}}}}} & (3)\end{matrix}$We note that importance weighted classification can be solved very wellwith a classification method, by use of weighted rejection samplingtechniques (see,again, Zadronzny, Langford, and Abe, ibid.).

Hypothesis Representations and Other Notation

In the above, we assumed that the hypotheses output by a cost-sensitivelearner is a functional hypothesis h, i.e., h:X→Y. It is also possibleto allow hypotheses that are stochastic, namelyh:X×Y→[0,1]subject to the stochastic condition:${\forall{x \in {X{\sum\limits_{y \in Y}{h\left( {x,y} \right)}}}}} = 1.$With stochastic hypotheses, stochastic cost-sensitive learning isdefined as that of minimizing the following expected cost:$\arg\quad{\min\limits_{h}{E_{D}{\sum\limits_{y \in Y}{{C\left( {x,y} \right)}{h\left( {x,y} \right)}}}}}$Note that in the special case that h is deterministic, this formulationis equivalent to the definition given in Equation (1). Also, this is aconvexification of the standard objective function that we usuallyexpect a stochastic cost-sensitive learner to minimize, i.e.,$E_{D}\left\lbrack {C\left( {x,{\arg\quad{\max\limits_{y \in Y}{h\left( {x,y} \right)}}}} \right)} \right\rbrack$We also consider a variant of cost-sensitive learning in whichrelational hypotheses are allowed. Here relational hypotheses h arerelations over X×Y, i.e., h:X×Y→{0, 1}. In general h is neitherfunctional nor stochastic, and in particular it may violate thestochastic condition,${\sum\limits_{y \in Y}{h\left( {x,y} \right)}} = 1.$

We often use the more general notation of h(x,y), meant for stochasticand relational hypotheses, even when h is a deterministic function fromX to Y. As notational shorthand, for a stochastic hypothesis h, we writeh(x) to denote h(x, ·):Y→[0,1], and C(x, h(x)) to denote the expectedcost of its predictions, i.e.,${C\left( {x,{h(x)}} \right)} = {\sum\limits_{y \in Y}{{h\left( {x,y} \right)}{{C\left( {x,y} \right)}.}}}$Finally, we note that we often write “x ε S” as a shorthand for “

y ε Y(x,y) ε S”.

The Methodology

Our methodology can be interpreted as a reduction, which translates amulti-class cost-sensitive learning problem to a classifier learningproblem. That is, it allows us to solve the cost-sensitive learningproblem using an arbitrary classifier learning method as a componentalgorithm. This methodology is derived using two key ideas: 1) expandingdata space and 2) gradient boosting with stochastic ensembles.Theoretical performance guarantee on a particular variant of theinvented methodology is derived using a convexification of the objectivefunction by the expected cost function. Below we will explain these twokey ideas by exhibiting a prototypical method based on each.

A representative method in the prior art of iterative methods forcost-sensitive learning is the method proposed in Zadrozny, Langford andAbe, ibid., called costing. The weighting scheme of this method exploitsthe following observation: For the binary class case, the aboveformulation in terms of cost per example, C(x, y₂), can be furtherreduced to a formulation in terms of a single importance number perexample. This is possible by associating a number indicating theimportance of an example (x, y₂), given by |C(x, 0)−C(x, 1)|. Thisconversion allows us to reduce the cost-sensitive learning problem to aweighted classifier learning problem, but it has not been known how thatwould be done for the multi-class scenario. It is therefore natural toconsider iterative weighting schemes, in which example weights areiteratively modified in search for the optimal weighting.

A straightforward application of iterative weighting suffers from aninability to directly take into account the different costs associatedwith multiple ways of misclassifying examples. This translates tonon-convergence of the method in practice. We address this issue by thetechnique of expanding data space, the first of the two key ideas.

Data Space Expansion

The objective of minimizing the empirical cost on the original trainingsample is equivalent to minimization on the following expanded sample.Given a labeled sample S consisting of (x, (C(x,y))_(yεY)) of size m, wedefine a sample S′ of size mk for classification, where k is the size ofthe label set, i.e., k=|Y|, as follows.$\left. {{S^{\prime} = {\left\{ {\left( {x,y} \right),{{\max\limits_{x,y}{C\left( {x,y} \right)}} - {C\left( {x,y} \right)}}} \right)❘{\left( {x,\left\langle {C\left( {x,y} \right)} \right\rangle_{y \in Y}} \right) \in S}}},{y \in Y}} \right\}$Minimizing the importance weighted loss,$\sum\limits_{{({x,y,c})} \in S^{\prime}}{c \cdot {I\left( {{h(x)} \neq y} \right)}}$on this new dataset also minimizes the cost on our original sample. Thealgorithm DSE (Data Space Expansion) takes advantage of thisobservation, which is summarized below as a theorem.

THEOREM 1. With the definitions given in FIG. 3, a hypothesis ILminimizing the weighted classification error on the expanded weightedsample S′,$\underset{{({x,y,c})}\sim S^{\prime}}{\hat{E}}\left\lbrack {c \cdot {I\left( {{h(x)} \neq y} \right)}} \right\rbrack$also minimizes the cost on the original sample S,${\underset{{({x,y,c})}\sim S^{\prime}}{\hat{E}}\left\lbrack C_{h{(x)}} \right\rbrack}.\begin{matrix}{\arg\quad{\min\limits_{h}{\underset{{({x,y,c})}\sim S^{\prime}}{\hat{E}}\left\lbrack {c \cdot {I\left( {{h(x)} \neq y} \right)}} \right\rbrack}}} \\{= {\arg\quad{\min\limits_{h}{\underset{{({x,\overset{\sim}{C}})}\sim S}{\hat{E}}{\sum\limits_{y \in Y}\left\lbrack {\left( {{\max\limits_{y^{\prime} \in Y}C_{y^{\prime}}} - C_{y}} \right) \cdot {I\left( {{h(x)} \neq y} \right)}} \right\rbrack}}}}} \\{= {\arg\quad{\max\limits_{h}{\underset{{({x,\overset{\sim}{C}})}\sim S}{\hat{E}}{\sum\limits_{y \in Y}\left\lbrack {C_{y} \cdot {I\left( {{h(x)} \neq y} \right)}} \right\rbrack}}}}} \\{= {\arg\quad{\max\limits_{h}{\underset{{({x,\overset{\sim}{C}})}\sim S}{\hat{E}}\left\lbrack {\left( {\sum\limits_{y \in Y}C_{y}} \right) - C_{h{(x)}}} \right\rbrack}}}} \\{= {\arg\quad{\max\limits_{h}{\underset{{({x,\overset{\sim}{C}})}\sim S}{\hat{E}}\left\lbrack C_{h{(x)}} \right\rbrack}}}}\end{matrix}$Gradient Boosting with Stochastic Ensembles

Having described the idea of data space expansion, we now cmbine it withthe gradient boosting framework to arrive at our main method. Inparticular, we cast the stochastic multiclass cost-sensitive learning inthe framework of gradient boosting (see L. Mason, J. Baxter, P. Barlett,and M. Frean, “Boosting algorithms as gradient descent”, Advances inNeural Information Processing Systems 12, pp. 512-518, 2000), with theobjective function defined as the expected cost of the stochasticensemble, obtained as a mixture of individual hypotheses, on theexpanded data set. As we stated above, a functional hypothesis of theform h:X→Y can be viewed as a special case of a stochastic hypothesis.We then define a stochastic ensemble hypothesis H, given multiplefunctional hypotheses, h_(t), t=1, . . . , T, as the conditionaldistribution defined as the mixture of the component hypotheses, namely,${\forall{x \in X}},{\forall{y \in Y}},{{H\left( {x,y} \right)} = {\sum\limits_{t = 1}^{T}{h_{t}\left( {x,y} \right)}}}$Let H_(t) denote the mixture hypothesis of the learning procedure atround t. The procedure is to update its current combined hypothesis bythe mixture of the previous combined hypothesis and a new hypothesis,i.e., by settingH _(t)(x,y)=(1−β)H _(t-1)(x,y)+βh(x,y)Thus, the expected cost of H_(t) on x isC(x, H _(t)(x))=(1−β)C(x, H _(t-1)(x))+βC(x, h _(t)(x))Now, suppose that h predicts a particular label y for x, i.e., h(x,y)=1,thenC(x, H _(t)(x))=(1−β)C(x, H _(t-1)(x))+βC(x,y)If we now take a derivative of this function with respect to β, we get$\frac{\partial{C\left( {x,{H_{t}(x)}} \right)}}{\partial\beta} = {{C\left( {x,y} \right)} - {C\left( {x,{H_{t - 1}(x)}} \right)}}$Note that this is the difference between the average cost of the currentensemble hypothesis and the new weak hypothesis assigning probabilityone to the specified label.

We then take this derivative with respect to all data points (x,y) inthe expanded data set S′, and thus the gradient is mk-dimensional. Wethen expect the weak learner to find a hypothesis h whose inner-productwith the negative gradient is large. That is, the output h of the weaklearner seeks to maximize the following sum. $\begin{matrix}{{- \left\langle {h,{\nabla C}} \right\rangle} = {\frac{1}{W}{\sum\limits_{x \in S}{\sum\limits_{y \in Y}{\left( {{C\left( {x,{H_{t - 1}(x)}} \right)} - {C\left( {x,y} \right)}} \right){h\left( {x,y} \right)}}}}}} & (9)\end{matrix}$where W denotes the sum of absolute values of the weights, i.e.,$W = {\sum\limits_{x \in S}{\sum\limits_{y \in Y}{{{{C\left( {x,{H_{t - 1}(x)}} \right)} - {C\left( {x,y} \right)}}}.}}}$Note that unlike the weights typically used in existing hosting methods,the weights w_(x,y):=C(x, H_(t-1)(x))−C(x,y) can be negative, since y isnot necessarily the best (least cost) label. This means that the weaklearner now receives both positive and negative weights. While theminimization of weighted misclassification with positive and negativeweights makes perfect sense as an optimization problem, itsinterpretation as a classification problem is not immediately clear. Inparticular, it prohibits the use of weighted sampling as a means ofrealizing the weighted classification problem.

We deal with this problem by converting a relational version of theweighted multi-class classification problem (i.e., of finding h tomaximize Equation 9) in each iteration to a weighted binaryclassification problem. Specifically, we convert each example pair (x,y)to ((x,y), l), and set l=1 if the weight on (x,y) is positive, and l=0if the weight is negative. The output hypothesis of the binaryclassifier is in general relational, so it is converted to a stochastichypothesis by the procedure Stochastic. (The particular way thisprocedure is defined is motivated by the theoretical guarantee, whichwill be shown in the next subsection.) The overall process, consistingof multiple iterations of such a reduction, constitutes a reduction ofthe stochastic multi-class cost-sensitive classification to binaryweighted classification.

With the foregoing definitions, we can now state our main method, GBSE(Gradient Boosting with Stochastic Ensembles).

Theoretical Performance Guarantee on a Variant

It turns out that a strong theoretical performance guarantee can beproved on a variant of this method, which we describe below. We definethe per label average cost, {tilde over (C)}(x, H(x)), of a stochastichypothesis H, in general, as follows.${\overset{\sim}{C}\left( {x,{H(x)}} \right)} = {\frac{1}{k}{\sum\limits_{y \in Y}{{H\left( {x,y} \right)}{C\left( {x,y} \right)}}}}$Note that, with this definition, the empirical loss (cost) of H on theoriginal sample S, C(H, S), can be expressed as the sum of this perlabel cost over the expanded data set S′={(x,y)|x ε S, y ε Y}.$\begin{matrix}{{C\left( {H,S} \right)} = {\sum\limits_{x}{\sum\limits_{y}{{H\left( {x,y} \right)}{C\left( {x,y} \right)}}}}} \\{= {\sum\limits_{x}{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H(x)}} \right)}}}}\end{matrix}$The variant, for which we prove our theoretical performance guarantee isobtained by simply replacing the weight updating rule of GBSE by thefollowing:w _(x,y) ={tilde over (C)}(x, H _(t-1)(x))−C(x,y)The resulting variant, which we call GBSE-T (Gradient Boosting withStochastic Ensembles—Theoretical version), is summarized in FIG. 5.

We can show that GBSE-T has a boosting property given a version of weaklearning condition on the component classifier. This weak learningcondition, which we make precise below, is one that is sensitive toclass imbalance.

DEFINITION 1. We say that an algorithm A for the binary importanceweighted classification problem, as defined above, satisfies the weaklearning condition for a given classification sample S=(x,y)^(m), if forarbitrary distribution over S, (w)^(m), Σw=1, when it is givenS′=(x,y,w)^(m) as input, its output h satisfies the following, for somefixed γ>0: $\begin{matrix}{{\sum\limits_{{({x,y,w})} \in S^{\prime}}{w \cdot {I\left( {{h(x)} = y} \right)}}} \geq {{\sum\limits_{y = 0}w} + {\gamma{\sum\limits_{y = 1}w}}}} & (12)\end{matrix}$

THEOREM 2. Suppose that the component leaner A satisfies the weaklearning condition for the input sample S. Then, the output of GBSE-Twill converge to a stochastic ensemble hypothesis achieving minimumexpected cost on the (original) sample S. In particular, if we set αt=αfor all t,${\sum\limits_{x}{\sum\limits_{y}{{H_{T}\left( {x,y} \right)}{C\left( {x,y} \right)}}}} \leq {\exp\left\{ {{- \frac{\gamma\alpha}{k}}T} \right\}{\sum\limits_{x}{\sum\limits_{y}{{H_{0}\left( {x,y} \right)}{C\left( {x,y} \right)}}}}}$Proof

We first establish the following simple correspondence between the weaklearning conditions on the relational multi-class classification problemthat we wish to solve in each iteration, and the binary classificationproblem that is given to the component algorithm to solve it.

DEFINITION 2. Let S be a weighted sample of the form S=(x,y,w)^(m),where weights w can be both positive and negative. Then define atransformed sample S′ from S by S′=((x,y), l, |w|)^(m) where l=I(w≧0).

-   -   1. The relational weighted multi-class classification problem        for S is to find a relational hypothesis h:X×Y→{0, 1} that        maximizes the following sum:        ${a\left( {h,S} \right)} = {\frac{1}{W}{\sum\limits_{{({x,y,w})} \in S}{{w \cdot {h\left( {x,y} \right)}}\quad{where}}}}$        $W = {\sum\limits_{{({x,y,w})} \in S}{{w}.}}$    -   2. The weighted binary classification problem for S′ is to find        a hypothesis h′:X×Y→{0, 1} that maximizes the following weighted        classification accuracy:        ${a^{\prime}\left( {h^{\prime},S^{\prime}} \right)} = {\frac{1}{W}{\sum\limits_{{({{({x,y})},l,{w}})} \in S^{\prime}}{{w} \cdot {I\left( {{h^{\prime}\left( {x,y} \right)} = l} \right)}}}}$

LEMMA 1. Assume the notation of Definition 2. Then, for arbitrary ε>0, hsatisfies the following condition on the relational multi-classclassification problem for S:a(h,S)≧εif and only if (the same) h satisfies the corresponding condition on thetransformed binary classification problem for S′:${a^{\prime}\left( {h,S^{\prime}} \right)} \geq {\frac{\sum\limits_{l = 0}{w}}{W} + \varepsilon}$Proof of Lemma 1 $\begin{matrix}{{W \cdot {a\left( {h,S^{\prime}} \right)}} = {\sum\limits_{{({{({x,y})},l,{w}})} \in S^{\prime}}{{w} \cdot {I\left( {{h\left( {x,y} \right)} = l} \right)}}}} \\{= {{\sum\limits_{w \geq 0}{w \cdot {I\left( {{h\left( {x,y} \right)} = 1} \right)}}} + {\sum\limits_{w < 0}{{- w} \cdot {I\left( {{h\left( {x,y} \right)} = 0} \right)}}}}} \\{= {{\sum\limits_{w \geq 0}{w \cdot {h\left( {x,y} \right)}}} + {\sum\limits_{w < 0}{- {w\left( {1 - {h\left( {x,y} \right)}} \right)}}}}} \\{= {{\sum\limits_{{({x,y,w})} \in S}{w \cdot {h\left( {x,y} \right)}}} + {\sum\limits_{w < 0}{w}}}} \\{= {{W \cdot {a\left( {h,S} \right)}} + {\sum\limits_{{({x,y,w})} \in {S:{w < 0}}}{w}}}}\end{matrix}$Hence the lemma follows.Proof of Theorem 2

First, note that applying Stochastic to h, can increase the expectedcost only for x's such that |[y|h_(t)(x,y)=1}|=0, and for such x's thecost of f, equals that of H{t-1} by the definition of Stochastic. Hence,the empirical cost of f, on the original sample S, C(f_(t), S),satisfies the following: $\begin{matrix}{{{C\left( {f_{t},S} \right)} - {C\left( {h_{t},S} \right)}} \leq {\sum\limits_{{x:{\forall{y\quad{h{({x,y})}}}}} = 0}{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}}} & (13)\end{matrix}$Now recall that the expected empirical cost of H_(t) equals thefollowing, where we drop the subscript t from α_(t). $\begin{matrix}\begin{matrix}{{C\left( {H_{t},S} \right)} = {{\sum\limits_{x,y}{\left( {1 - \alpha} \right){H_{t - 1}\left( {x,y} \right)}{C\left( {x,y} \right)}}} + {\alpha{\sum\limits_{x,y}{{f\left( {x,y} \right)}{C\left( {x,y} \right)}}}}}} \\{= {{\sum\limits_{x,y}{\left( {1 - \alpha} \right){\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}} + {\alpha{\sum\limits_{x,y}{{f\left( {x,y} \right)}{C\left( {x,y} \right)}}}}}}\end{matrix} & (14)\end{matrix}$Hence, by combining Equation 13 and Equation 14, we can show thefollowing bound on the decrease in empirical cost in each iteration:C(H_(t − 1), S) − C(H_(t), S) $\begin{matrix}{\quad{= {\sum\limits_{x}{\alpha\left( {{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}} - {\sum\limits_{y}{{f\left( {x,y} \right)}{C\left( {x,y} \right)}}}} \right)}}}} \\{= {{\sum\limits_{x}{\alpha\left( {{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}} - {\sum\limits_{y}{{h\left( {x,y} \right)}{C\left( {x,y} \right)}}}} \right)}} +}} \\{{\sum\limits_{x}{\alpha\left( {{\sum\limits_{y}{{h\left( {x,y} \right)}{C\left( {x,y} \right)}}} - {\sum\limits_{y}{{f\left( {x,y} \right)}{C\left( {x,y} \right)}}}} \right)}} \geq} \\{{\sum\limits_{x}{\alpha\left( {{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}} - {\sum\limits_{y}{{h\left( {x,y} \right)}{C\left( {x,y} \right)}}}} \right)}} -} \\{{\alpha\left( {\sum\limits_{{x:{\forall{y\quad{h{({x,y})}}}}} = 0}{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}} \right)} \geq} \\{\alpha\left( {\sum\limits_{x}\left( {{\sum\limits_{{y:{h{({x,y})}}} = 1}{{h\left( {x,y} \right)}\left( {{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)} - {C\left( {x,y} \right)}} \right)}} +} \right.} \right.} \\{{\sum\limits_{{y:{h{({x,y})}}} = 0}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}} - {\sum\limits_{{x:{\forall{y\quad{h{({x,y})}}}}} = 0}{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}}} \\{= {\alpha\left( {{\sum\limits_{x}\left( {{\sum\limits_{y}{{h\left( {x,y} \right)}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}} - {C\left( {x,y} \right)}} \right)} +} \right.}} \\{\left. \left( {{\sum\limits_{x}{\sum\limits_{{y:{h{({x,y})}}} = 0}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}} - {\sum\limits_{{x:{\forall{y\quad{h{({x,y})}}}}} = 0}{\sum\limits_{y}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}}} \right) \right) \geq} \\{\left. {{\alpha{\sum\limits_{x}{\sum\limits_{y}{{h\left( {x,y} \right)}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}}}} - {C\left( {x,y} \right)}} \right) \geq} \\{{{\alpha\underset{x}{\gamma\sum}{\sum\limits_{y:{{{\overset{\sim}{C}{({x,{H_{t - 1}{(x)}}})}} - {C{({x,y})}}} > 0}}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}}} - {C\left( {x,y} \right)}} \geq} \\{{\underset{x}{{\alpha\gamma}\sum}{\overset{\sim}{C}\left( {x,{H_{t - 1}(x)}} \right)}} = {\frac{\alpha\gamma}{k}{C\left( {H_{t - 1},S} \right)}}}\end{matrix}$In the above derivation, the second to last inequality follows from theweak learning condition and applying Lemma 1 with weights {tilde over(C)}(x, H_(t-1)(x))−C(x,y). The last inequality follows from the factthat the weights are nonnalized so that the minimum achievable cost iszero for all x. Noting that the sum of these weights is positivewhenever the current ensemble hypothesis is sub-optimal, this guaranteesa positive progress in each iteration unless optimality is achieved.Since the expected empirical cost function as defined by Σ_(x)Σ_(y)F(x,y) C(x,y) is convex (in fact linear), this implies convergence tothe global optimum. Noting that in each iteration, the empirical cost isreduced at least by a factor of ${1 - \frac{\gamma\alpha}{k}},$and the theorem follows.

Note that at earlier iterations, the binary classifier used as thecomponent learner is likely to be given weighted sample with balancedpositive and negative examples. As the number of iterations increasesand progress is made, however, it will receive samples that areincreasingly more negative. (This is because the positive examplescorrespond to labels that can further improve the current performance.)It therefore becomes easier to attain high weighted accuracy by simplyclassifying all examples to be negative. The weak learning condition ofEquation 12 appropriately deals with this issue, as it requires that theweak learner achieve better weighted accuracy than that attainable byassigning all examples to the negative class.

Variations

In addition to the two variants of the Gradient Boosting withStochastiEnsembles method presented above, namely GBSE and GBSE-T, otherrelated variations are possible. For example, in one variant, theweighted sampling can be done in two steps; the instance is sampled inthe first step according to a probability proportional tomax_(y)×w_(x,y)and then choosing the label y with a probability proportional to|w_(x,y)|In a yet another variant, the weighted sampling can be done in twosteps; the instance is sampled in the first step according to the sameprobability as above, and for the chosen instance, examples aredeterministically added for all possible labels.

Implementation

Referring now to FIG. 1, there is shown a system on which a method formulti-class, cost-sensitive learning according to the invention may beimplemented. This system comprises a multi-class cost-sensitive learningtop control module 1 which controls the overall control flow, making useof various sub-components of the system. A learning algorithm storagemodule 2 stores a representation of an algorithm for classificationlearning. An arbitrary algorithm for classification can be used here.Alternatively the learning algorithm can be a decision tree learningalgorithm, a naïve Bayes method, a logistic regression method or neuralnetworks. The model output module 3 stores the models obtained as aresult of applying the learning algorithm stored in module 2 to trainingdata given by weighted sampling module 4 and outputs a final model byaggregating these models. The weighted sampling module 4 accesses thedata stored in data storage module 7, samples a relatively small subsetof the data with acceptance probability determined using the exampleweights, and passes the obtained sub-sample to module 1. The weightupdate module 5 updates the example weights for sampling using aparticular function determined by the current weights and currentmodels. The model update module 6 updates the current model using themodel's output in the previous iterations stored in the current modelstorage module 8 and the output model of the current iteration output bymodule 3 and stores the resulting updated model in module 8.

FIG. 2 shows a flow diagram of the process implemented in the system ofFIG. 1. The first three steps initialize the process. In Step 21,expanded data T is initialized using the input data S. In Step 22, H₀ isinitialized by setting for all (x,y) in T. Finally, in Step 23, theweights for all (x,y) in Tare initialized. The iteration begins in thedecision block of Step 4. A test is made to determine if i=t. If not,Step 25 performs the computation for all (x,y)${w\left( {x,y} \right)} = {\left( {\sum\limits_{y \in Y}{{H_{t - 1}\left( {x,y} \right)} \cdot {C\left( {c,y} \right)}}} \right) - {C\left( {x,y} \right)}}$The decision block in Step 26 determines if there is more data in T or aSTOP condition has been met. If not, in Step 27, (x,y) is sampled from Tand accepted with a probability proportional to |w(x,y)|. Next, in Step28, if accepted, ((x,y), |(w(x,y)>0)) is added to sub-sample T^(t). Areturn is then made to the decision block in Step 26. When there is nomore data in T or a STOP condition has been met, the process goes toStep 29 where the learning algorithm is run on T^(t) to obtain modelh_(t). Next, in Step 30, f_(t) is set equal to stoch(h_(t)). Then, inStep 31, α_(t) is chosen and H_(t) is set equal to(1−α_(t))H_(t-1)+α_(t)f_(t). The index i is incremented at Step 31, anda return is then made to the decision block in Step 24. If i=t, then inStep 33 the final model H_(t) is output.

FIG. 3 shows a system on which another method for multi-class,cost-sensitive learning according to the invention may be implemented.This system is similar to that shown in FIG. 1 and comprises amulti-class cost-sensitive learning top control module 1 which controlsthe overall control flow, making use of various sub-components of thesystem, a learning algorithm storage module 2, which stores arepresentation of an algorithm for classification learning, a modeloutput module 3, which stores the models obtained as a result ofapplying the learning algorithm stored in module 2 to training datagiven by weighted sampling module 4 and outputs a final model byaggregating these models, and a weighted sampling module 4, whichaccesses the data stored in data storage module 7, samples a relativelysmall subset of the data with acceptance probability determined usingthe example weights, and passes the obtained sub-sample to module 1. Theweight calculation module 5′ replaces the weight upadate module 5, whichupdates the example weights for sampling using a dynamically changingweighting scheme. The model update module 6 updates the current modelusing the model's output in the previous iterations stored in thecurrent model storage module 8 and the output model of the currentiteration output by module 3 and stores the resulting updated model inmodule 8.

FIG. 4 shows a flow diagram of the process implemented in the system ofFIG. 3. The first step initializes the process. In Step 41, expandeddata T is initialized using the input data S. In Step 42, the weightsfor all (x,y) in Tare set. The iteration begins in the decision block ofStep 43. A test is made to determine if i=t. If not, a test is made inStep 44 to determine if there is no more data in T or a stop conditionhas been met. If not, Step 45 samples (x,y) from T and accepts (x,y)with probability proportional to w(x,y). If accepted, (x,y) is added tosub-sample T′ in Step 46. The process then loops back to decision blockin Step 44 until there is either no more data in T or a stop conditionhas been met. At this point, the learning algorithm is run in Step 47 onT′ to obtain a model h_(t). In Step 48, α is chosen so that when i=1,α=0 and H_(t)=(1−α_(t))H_(t-1)+α_(t)f_(t). The index i is incremented atStep 49, and a return is then made to the decision block in Step 43. Ifi=t, then in Step 50 the final model H_(t) is output.

As a concrete example of applying the method of the invention to a realworld problem, we describe an application to network intrusiondetection. Network intrusion detection has recently become aproto-typical application problem for multi-class, cost-sensitivelearning. The multi-class aspect is essential because in thisapplication there are typically more than one level of intrusiondetection, such as probing and denial of service. The cost-sensitiveaspect is important because vastly different costs are associated withdifferent types of misclassification (e.g., false negatives are usuallya magnitude more costly than false positives) and it is absolutelycritical that any learning method used to derive an intrusion detectionrule is sensitive to this cost structure.

A network intrusion detection system based on the method and system ofthe invention for multi-class, cost-sensitive learning consists of thefollowing steps:

-   -   1) Convert past network connection data to a set of feature        vectors, by mapping information on a network connection to a        feature vector.    -   2) Label each of these vectors with known labels, such as        “normal”, “probe”, “denial of service”, or specific types of        intrusions.    -   3) Apply the method of the invention on the above data set, and        obtain a classification rule.    -   4) Convert new network connection data to feature vectors, apply        the above classification rule to them, and flag those        connections corresponding to feature vectors that are classified        as different types of “intrusions” as such.

A typical set of features used to transform connection data into awell-defined feature vector is that used in the network intrusion dataset known as “KDD CUP 99” data, which is publically available. Here isthe list of features in this data set (given in three separate tables).Basic Features of Individual TCP Connections feature name descriptiontype duration length (number of seconds) of the continuous connectionprotocol_type type of protocol, e.g., TCP, UDP, etc. discrete servicenetwork service on the destination, e.g., discrete http, telnet, etc.src_bytes number of data bytes from source to continuous desitinationdst_bytes number of data bytes from destination to continuous sourceflag normal or error status of the connection discrete land 1 ifconnection is from/to the same discrete host/port; 0 otherwisewrong_fragment number of “wrong” fragments continuous urgent number ofurgent packets continuous

Content Features Within a Connection Suggested by Domain Knowledgefeature name description type hot number of “hot” indicators continuousnum_failed_logins number of failed login attempts continuous logged_in 1if successfully logged in; 0 discrete otherwise num_compromised numberof “compromised” conditions continuous root_shell 1 if root shell isobtained; 0 discrete otherwise su_attempted 1 if “su-root” commandattempted; 0 discrete otherwise num_root number of “root” accessescontinuous num_file_creations number of file creation operationscontinuous num_shells number of shell prompts continuousnum_access_files number of operations on access continuous control filesnum_outbound_cmds number of outbound commands in an continuous ftpsession is_hot_login 1 if the login belongs to the “hot” discrete list;0 otherwise is_guest_login 1 if the login is a “guest” login; 0 discreteotherwise

Traffic Features Computed Using a Two-Second Time Window feature namedescription type count number of connections to the same host continuousas the current connection in the past two seconds Note: The followingfeatures refer to these same host connections. serror_rate % ofconnections that have “SYN” continuous errors rerror_rate % ofconnections that have “REJ” continuous errors same_srv_rate % ofconnections of the same service continuous diff_srv_rate % ofconnections of different services continuous srv_count number ofconnections to the same continuous service as the current connection inthe past two seconds Note: The following features refer to thesesame-service connections. srv_server_rate % of connections that have“SYN” continuous errors srv_rerror_rate % of connections that have “REJ”continuous errors srv_diff_host_rate % of connections to different hostscontinuous

As a result of applying the multi-class, cost-sensitive learning methodof the invention to a data set consisting of these features and thecorresponding labels, using a decision tree algorithm as the“classification learning algorithm” stored in Module 2 of FIG. 1, oneobtains, as the classification rule, a voting function over a number ofdecision trees, such as the tree shown in FIG. 5.

The system diagram of FIG. 1 and the flow chart of FIG. 2 illustrate apreferred embodiment of the invention, which corresponds to the method“GBSE” described herein. However, it will be understood by those skilledin the art that the method “DSE”, also described herein, may be used inthe alternative. The main difference between DSE and GBSE is that inDSE, the sampling weights remain unchanged throughout all iterations.Consequently, the modules and funcationalities that are related toweight updating are unnecessary.

Experimental Evaluation

We use the C4.5 decision tree learner described by J. Quinlan in C4.5:Programs for Machine Learning, Morgan Kaufmann (1993), as the baseclassifier learning method, because it is a standard for empiricalcomparisons and it was used as the base learner by Domingos for theMetaCost method (see, P. Domingos, “MetaCost: A general method formaking classifiers cost sensitive”, Proceedings of the FifthInternational Conference on Knowledge Discovery and Data Mining, pp.155-164, ACM Press, 1999).

We compare our methods against three representative methods: Bagging(see L. Breiman, “Bagging predictors”, Machine Learning, 24(2):123-140,1996), Averaging cost (see, P. Chan and S. Stolfo, “Toward scalablelearning with non-uniform class and cost distributions”, Proceedings ofthe Fourth International Conference on Knowledge Discovery and DataMining, pp. 164-168, 1998), and MetaCost (see, Domingos, ibid.). TheAveraging cost method was also used for comparison in Domingos, ibid.Note that Bagging is a cost-insensitive learning method. Here we give abrief description of these methods, and refer the reader to Breiman,ibid., and Domingos, ibid., for the details.

-   -   Bagging obtains multiple sub-samples by sampling with        replacement, feeds them to the base learner (C4.5), and takes        the average over the ensemble of output hypotheses.    -   Averaging Cost (AvgCost) obtains a subsample by weighted        sampling with weights defined as the average cost for each x,        and then feeds it to the base learner (C4.5).    -   MetaCost uses bagging to obtain an ensemble of hypotheses, uses        the ensemble to estimate the class probabilities, and then        outputs a hypothesis that minimizes the expected risk with        respect to these estimates.

There are some deviations from these methods in our implementation,which we clarify below. The main deviation is that we use rejectionsampling for all methods, while other sampling schemes such asresampling with replacement are used in the original methods. We do thisfor two reasons: (1) inadequacy of resampling with replacement,especially for C4.5, has been noted by various authors (see, forexample, B. Zadrozny, J. Langford, and N. Abe, “Cost-sensitive learningby cost-proportionate example weighting”, Proceedings of the Third IEEEInternational Conference on Data Mining, pp. 435-442, 2003); and (2)since our methods use rejection sampling, we do the same for the othermethods for fairness of comparison. We stress that this deviation shouldonly improve their performance. Another deviation is that we use avariant of MetaCost that skips the last step of learning a classifier ona relabeled training data set. It has been observed that this variantperforms at least as well as MetaCost, in terms of cost minimization.(This variant has been called BagCost by D. Margineantu in Methods forCost-Sensitive Learning, PhD thesis, Department of Computer Science,Oregon State University, Corvallis, Oreg., 2001.) Also, in ourimplementation of AvgCost, we perform weighted sampling multiple timesto obtain an emsemble of hypotheses, then output their average as thefinal hypothesis. We note that, due to our normalization assumption thatthe minimum cost for each instance x is always zero, our version ofAvgCost is identical to a more sophisticated variant in which thedifference between the average cost and the minimum cost is used forsampling weights. Our experience shows that this variant of AvgCostperforms better than the original method.

The methods were applied to five benchmark datasets available from theUCI machine learning repository (C. L. Blake and C. J. Merz, “UCIrepository of machine learning databases”, Department of Information andComputer Sciences, University of California, Irvine, Calif., 1998) andone dataset from the UCI KDD archive (S. D. Bay, “UCI archive”,Department of Information and Computer Sciences, University ofCalifornia, 2000). These datasets were selected by the criteria ofhaving approximately 1,000 data or more, besides being multiclassproblems. A summary of these datasets is given in Table 1. TABLE 1 Dataset characteristics: data size, number of classes, and the ratio betweenthe frequency of the most common class to the least common. Dataset # ofexamples # of classes Class ratio Annealing 898 5 0.01316 KDD-99 1977105 0.0001278 Letter 20000 26 0.9028 Satellite 6435 6 0.4083 Solar flare1389 7 0.002562 Splice 3190 3 0.4634Except for the KDD-99 dataset, these datasets do not have standardmisclassification costs associated with them. For this reason, we followDomingos and generate cost matrices according to a model that giveshigher costs for misclassifying a rare class as a frequent one, andinversely for lowest. (Note therefore that our experiments do notexploit the full generality of the instance-dependent cost formulationpresented above.) This reflects a situation that is found in manypractical data mining applications, including direct marketing and frauddetection, where the rare classes are the most valuable to identifycorrectly.

Our cost model is as follows: Let {circumflex over (P)}(y₁) and{circumflex over (P)}(y₂) be the empirical probabilities of occurrenceof classes y₁and y₂ in the training data. We choose the non-diagonalentries of the cost matrix C(y₁, y₂), y₁≠y₂ with uniform probabilityfrom the interval [0,2000 {circumflex over (P)}(y₁)/{circumflex over(P)}(y₂)]. In Domingos, ibid., the diagonal entries were then chosenfrom the interval [0,1000], which often leads to cost matrices in whichthe correct label is not the least costly one. Besides beingunreasonable (see C. Elkan, “Magical thinking in data mining: Lessonsfrom coil challenge 2000”, Proceedings of the Seventh InternationalConference on Knowledge Discovery and Data Mining, pp. 426-43 1, ACMPress, 1999), these cost matrices can give an unfair advantage tocost-sensitive methods over cost-insensitive ones. We therefore set thediagonal entries to be identically zero, which is consistent with ournormalization assumption.

In all experiments, we randomly select ⅔ of the examples in the datasetfor training and use the remaining ⅓ for testing. Also, for eachtraining/test split we generate a different cost matrix according to therules above. Thus, the standard deviations that we report reflect bothvariations in the data and in the misclassification costs.

We remark on certain implementation details of the proposed learningmethods in our experimentation. First, we note that in all of themethods used for comparison, C4.5 was used as the component algorithm,and the final hypothesis is expressed as an ensemble of output decisiontress of C4.5. Its output hypothesis is therefore also an ensemble ofdecision trees. Next, the choice of the mixture weight α_(t) wasunspecified in the algorithm descriptions. The choice of α_(t) was setat 1/t for most methods.

The results of these experiments are summarized in Tables 2 and 3. TABLE2 Experimental results: the average cost and standard error. DatasetBagging AvgCost MetaCost DSE GBSE Annealing  1059 ± 174 127.4 ± 12.2206.8 ± 42.8 127.1 ± 14.9 33.72 ± 4.29 Solar  5403 ± 397 237.8 ± 37.5 5317 ± 390 110.9 ± 28.7 48.17 ± 9.52 KDD-99 319.4 ± 42.2 42.43 ± 7.9549.39 ± 9.34 46.68 ± 10.16  1.69 ± 0.78 letter 151.0 ± 2.58 91.90 ± 1.36129.6 ± 2.44 114.0 ± 1.43 84.63 ± 2.24 Splice 64.19 ± 5.25 60.78 ± 3.6549.95 ± 3.05 135.5 ± 14 57.50 ± 4.38 Satellite 189.9 ± 9.57 107.8 ± 5.95104.4 ± 6.43 116.8 ± 6.28 93.05 ± 5.57

TABLE 3 Experimental results: the average data size used by each methodin 30 iterations, and standard error. Dataset Bagging AvgCost MetaCostDSE GBSE Annealing  11991 ± 13.1  1002.8 ± 183  11987 ± 9.84  3795.5 ±688  1260.2 ± 224 Solar  18499 ± 20.4  334.80 ± 37.5  18510 ± 14.4 2112.8 ± 276  486.45 ± 53.3 KDD-99 395310 ± 143  2551.9 ± 428.6 395580± 143   12512 ± 2450   4181 ± 783.6 letter  40037 ± 44.3  159720 ± 2028 40052 ± 41  479130 ± 2710  363001 ± 5557 Splice  42515 ± 26.6   33658 ±1697  42501 ± 21   52123 ± 592   50284 ± 3659 Satellite  86136 ± 123  60876 ± 1641  85984 ± 127  218870 ± 6516  140810 ± 3335

Table 2 lists the average costs attained by each of these methods on the6 data sets, and their stand errors. These results were obtained byaveraging over 20 runs, each run consisting of 30 iterations of therespective learning method. These results appear quite convincing: GBSEout-performs all comparison methods on all data sets, except on Splice,for which it ranks second after MetaCost. Also, GBSE is the bestperforming among the proposed methods, confirming our claim that thecombination of various techniques involved is indeed necessary to attainthis level of performance.

Table 3 lists the average total data size used by each of the methods in30 iterations. Examining these results in conjunction with the datacharacteristics in Table 1 reveals a definite trend. First, note thatthe data sets are divided into to groups: those having very large skews,or very low class ratios (Annealing, KDD-99 and Solar flare), and thosehaving moderate skews (Satellite, Splice and Letter). It is evident thatthe methods based on example weighting (AvgCost, GBSE, DSE) usemagnitudes smaller data sizes for the three data sets in the first group(i.e., with large skews), as compared to other methods, Bagging andMetaCost. The performance of GBSE is especially impressive on thisgroup, achieving much lower costs while requiring very small data sizes.It is worth mentioning that it is these data sets in the first groupwith large skews that require cost-sensitive learning the most.

We have provided a novel method for multiclass cost-sensitive learningbased on gradient boosting with stochastic ensembles. It is not thefirst time that the issue of incorporating cost-sensitivity to boostinghas been addressed. For example, AdaCost (see W. Fan, S. J. Stolfo, J.Zhang, and P. K. Chan, “AdaCost: Misclassification cost-sensitiveboosting”, Proceedings of the Sixteenth International Conference onMachine Learning, pp. 97-105, 1999) suggested a way of modifyingAdaBoost's exponential loss using a function (called cost adjustmentfunction) of the cost and confidence. The rational choice of this costadjustment function, however, appears not to be well-understood. Thestochastic ensemble that we employ in this method provides astraightforward but reasonable way of incorporating cost and confidence;i.e., in terms of expected cost.

While the invention has been described in terms of a single preferredembodiment, those skilled in the art will recognize that the inventioncan be practiced with modification within the spirit and scope of theappended claims.

1. A method for multi-class, cost-sensitive learning based on iterative example weighting schemes applied to a chosen data set comprising the steps of: a) obtaining an expanded data set, which is defined by enhancing each example in an original data set with as many data points as there are possible labels for any single instance; b) repeatedly drawing sub-samples from the expanded data set using weighted sampling according to a certain example weighting scheme that remains constant throughout the iterations, in which each labeled example is given the weight specified as the difference between the maximum possible misclassification cost for the instance in question and the misclassification associated with the label in the particular labeled example; c) calling a component classification learning algorithm to the sub-sample obtained in step b) and obtaining a hypothesis representing a classifier; d) outputting all classifier representations obtained through the iterations and representing an average over them, each of which can be an arbitrary representation of classifier for a problem at hand; and e) outputting all of the representations obtained through the iterations representing an average over them, each of which can be an arbitrary representation of classifier for the problem at hand.
 2. The method for multi-class, cost-sensitive learning recited in claim 1, wherein the learning algorithm is an arbitrary algorithm for classification.
 3. The method for multi-class, cost-sensitive learning recited in claim 1, wherein the learning algorithm is selected from the group consisting of decision tree algorithms, naïve Bayes method, logistic regression method and neural networks.
 4. A method for multi-class, cost-sensitive learning based on an example weighting scheme applied to a chosen data set comprising the steps of: a) obtaining an expanded data set, which is defined by enhancing each example in an original data set with as many data points as there are possibles for any single instance; b) iteratively applying weighted sampling from the expanded data set, using a dynamically changing weighting scheme involving both positive and negative weights; c) calling a component classification algorithm on a modified binary classification problem in which each example is itself already a labeled pair, and its (meta) label is 1 or 0 depending on whether the example weight in the above weighting scheme is positive or negative, respectively, and obtains a hypothesis representing a classifier; d) optionally modifying the obtained classifier, which is in general a relation on the original classification (mapping elements of domain to the labels, so that it is stochastic, namely a conditional probability distribution so that its probabilities over the set of labels sum to one for each instance; e) outputting all representations obtained through the iterations and representing an average over them, each of which can be an arbitrary representation of classifier for the problem at hand.
 5. The method for multi-class, cost-sensitive learning recited in claim 4, wherein the learning algorithm is an arbitrary algorithm for classification.
 6. The method for multi-class, cost-sensitive learning recited in claim 4, wherein the learning algorithm is selected from the group consisting of decision tree algorithms, naïve Bayes method, logistic regression method and neural networks.
 7. The method for multi-class, cost-sensitive learning recited in claim 4, wherein the dynamically changing weighting of step b) gives each labeled example a weight specified as a difference between an average cost on that instance by the averaged hypotheses from iterations so far and a misclassification cost associated with the label in the labeled example in question
 8. The method for multi-class, cost-sensitive learning recited in claim 4, wherein the dynamically changing weighting of step b) gives each labeled example a weight specified as a difference between an average cost on that instance by an averaged hypotheses from iterations so far divided by a number of labels per instance, and a misclassification cost associated with the label in the labeled example in question.
 9. The method for multi-class, cost-sensitive learning recited in claim 4, wherein the dynamically changing weighting of step b) gives each labeled example a weight specified as a difference between an average cost on that instance by an averaged hypotheses from iterations so far divided by a number of labels per instance, and a misclassification cost associated with the label in the labeled example in question, and the weighted sampling comprises the steps of: sampling the instance in step a) according to a probability proportional to a maximum of weights for that instance and any of the labels, and choosing a label with a probability proportional to the absolute value of a weight for that instance and the label in question.
 10. The method for multi-class, cost-sensitive learning recited in claim 4, wherein the dynamically changing weighting of step b) gives each labeled example a weight specified as a difference between an average cost on that instance by an averaged hypotheses from iterations so far divided by a number of labels per instance, and a misclassification cost associated with the label in the labeled example in question, and the weighted sampling comprises the steps of: sampling the instance step a) according to a probability proportional to a maximum of weights for that instance and any of the labels, and for the chosen instance, deterministically added examples for all possible labels.
 11. A system implementing a method for multi-class, cost-sensitive learning based on iterative example weighting schemes applied to a chose data set comprising: a multi-class cost-sensitive learning top control module controlling the overall control flow; a learning algorithm storage module storing a representation of a learning algorithm for classification learning; a model output module storing models obtained as a result of applying the learning algorithm to training data given by a weighted sampling module and outputting a final model by aggregating these models, said weighted sampling module accessing data stored in a data storage module, sampling a relatively small subset of the data with acceptance probability determined using the example weights, and passing the obtained sub-sample to said top control module; a weight calculation module updating the example weights for sampling using weighted sampling according to a weighting scheme that remains constant throughout iterations, in which each labeled example is given a weight specified as a difference between a maximum possible misclassification cost for the instance in question and a misclassification cost associated with the label in the particular labeled example; and a model update module updating current models using a model's output in previous iterations stored in a current model storage module and an output model of a current iteration output and storing a resulting updated model in said current storage module.
 12. The system for implementing a method for multi-class, cost-sensitive learning recited in claim 11, wherein the learning algorithm is an arbitrary algorithm for classification.
 13. The system for implementing a method for multi-class, cost-sensitive learning recited in claim 11, wherein the learning algorithm is selected from the group consisting of decision tree algorithms, naïve Bayes method, logistic regression method and neural networks.
 14. A system implementing a method for multi-class, cost-sensitive learning based on an example weighting scheme applied to a chosen data set comprising: a multi-class cost-sensitive learning top control module controlling the overall control flow; a learning algorithm storage module storing a representation of a learning algorithm for classification learning; a model output module storing models obtained as a result of applying the learning algorithm to training data given by a weighted sampling module and outputting a final model by aggregating these models, said weighted sampling module accessing data stored in a data storage module, sampling a relatively small subset of the data with acceptance probability determined using the example weights, and passing the obtained sub-sample to said top control module; a weight calculation module calculating the example weights for sampling using a dynamically changing weighting scheme involving both positive and negative weights; and a model update module updating the current models using the models output in the previous iterations stored in a current model storage module and the output model of the current iteration output and storing the resulting updated model in said current storage module.
 15. The system for implementing a method for multi-class, cost-sensitive learning recited in claim 14, wherein the learning algorithm is an arbitrary algorithm for classification.
 16. The system for implementing a method for multi-class, cost-sensitive learning recited in claim 14, wherein the learning algorithm is selected from the group consisting of decision tree algorithms, naïve Bayes method, logistic regression method and neural networks.
 17. The system implementing a method for multi-class, cost-sensitive learning recited in claim 14, wherein the dynamically changing weighting scheme used by the weight calculation module gives each labeled example a weight specified as a difference between an average cost on that instance by the averaged hypotheses from iterations so far and a misclassification cost associated with the label in the labeled example in question
 18. The system implementing a method for multi-class, cost-sensitive learning recited in claim 14, wherein the dynamically changing weighting scheme used by the weight calculation module gives each labeled example a weight specified as a difference between an average cost on that instance by an averaged hypotheses from iterations so far divided by a number of labels per instance, and a misclassification cost associated with the label in the labeled example in question.
 19. The system implementing a method for multi-class, cost-sensitive learning recited in claim 14, wherein the dynamically changing weighting scheme used by the weight calculation module gives each labeled example a weight specified as a difference between an average cost on that instance by an averaged hypotheses from iterations so far divided by a number of labels per instance, and a misclassification cost associated with the label in the labeled example in question, and the weighted sampling samples the instance according to a probability proportional to a maximum of weights for that instance and any of the labels, and a label is chosen with a probability proportional to the absolute value of a weight for that instance and the label in question.
 20. The system implementing a method for multi-class, cost-sensitive learning recited in claim 14, wherein the dynamically changing weighting scheme used by the weight calculation module gives each labeled example a weight specified as a difference between an average cost on that instance by an averaged hypotheses from iterations so far divided by a number of labels per instance, and a misclassification cost associated with the label in the labeled example in question, and the weighted sampling samples the instance according to a probability proportional to a maximum of weights for that instance and any of the labels, and for the chosen instance, deterministically adds examples for all possible labels. 